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The Behavior Analysis (SA) appj&oach to Project f Foilow 
Through, a federally funded education intervention program/ has 
reversed the trend cf academic failure of poor children by improving 
tie educational experience of poor children from 12 communities in 
the urban East/ Hidwest/ rural South # and on Indian reservations in. 
• - the* lest. The 3A model is supported by a multifaceted evaluation 
strategy/ predicated on four systematic process and outcome 
evaluation techniques* Each* serves a unigue function/ is used on a 
continuous and regular basis/ and is designed to complement the 
.others* The first strategy/ continuous progress assessment/ jls a 
computer-based system for monitoring the weekly academic progress of 
each child and classroom toward prescribed* year-end our^iclulm goals. 
Tie second evaluation technique is the annual testing orB^chil4££? 
and a comparable sasple of non~3A children on a standardized 
achievement test, fhe third strategy calls for regular formal and 
informal , observations of the classroom which allows the determination 
of^possbile needs for additional and/or specialized training of the 
classroom staff. The fourth evaluation measure is a survey of \ 
cc^nsuJ^r satisfaction of the program and its effects w&ich provides a 
• measure^f the social validity of the program. (Author) 
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Project folio* Through ~* 

Project Follow Through, formally established urtizt the Eco- 
nomic Opportunity Act in 1963 as part of the war on poverty, is 3 
federally-funded educational intervention program for early elenen 
tary grade school children (£g-3). Although Follow Through has 
~ no received the publicity of nost poverty programs , it h*s been 
and continues to be one of the msre successful programs. 

Tne intent of the program, as originally conceived, was t£ 
provide comprehensive educational, medical and social services to 
poor children while simultaneously involving parents directly in 
tfee progr&jn planning and decision-making processes* The plan of 
implementation for the educational component, known as ^planned 
variation,* was based on the assumption that no tingle instruc- 
tional approach was appropriate to all situations; .therefore a 
number of diverse, experimental educational models would be imple- 

■a 

nented and evaluated. Follow Through has been acclaimed as the 
largest social experiment -ever launched, and described as the 
nation *s largest research and evaluation program which has devel- 
* oped approaches to early eleaeptary conipensatory education. 

The Follow Through program , under the planned variation con- 
cept, consists of several educational raodels, developed and guided 
"by~a sponsor or sponsoring agency. A program sponsor has been 
defined as a professio'fcl person, an educator or psychologist, usu 
/ally associated with a university who develops a curriculua, a set 
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of teacher- training procedures, end takes responsibility fcr seeing 
that the procedures go into effect in a given set of classrooms 
(Kacoby and Zellner, 1970, p. 5). 

Conciiunities which applied for and received funds for follow 
Through clas^roorcs were allowed to Select tfce sponsor (or iroiel ) 
which appealed to tfre&; nost chose to affiliate with a sponsor , 
however, a few chose to be self- sponsored, inplenfenting and mas- . 
aoxrj cheir own educational approach. Currently there are twest^- 
'Follow Through sponsors annually serving r»earlj £0,000 cfriidngt in 
over 153 school, districts throughout the United States. / - * 

i 

The philosophy of planned variation coupled *?th extensive 

longitudinal evaluation has niade Project Follow Through a s^gntf- 

„ \-* 

icant and innovative compensatory educational program. Together 

with the outside evaluations performed for the Office of Education 

by independent research agencies, many Sponsors also perform in- 

» 

house process and outcome evaluations of the effect? of their taodel 
iinpleipentation strategies-. This paper will discuss the tnulti- 
face t e d -evaluation strategy, of one -of the Folliaw Tiiroitgh sp&nsorsr- 
^avior Analysis (8A).. 
Behavior Analysis Follow Through 

The Behavior Analysis program, located at the University of , 
Kansas (K.U.) initially began its role as sponsor in 1968 working 
with five cosrainities. . At present EA 'works with fifteen projects 
in twelve communities. Of the- Follow Through sponsors, BA is one 



of the irore widely dispersed programs with projects in the raral 
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South, the industrial fitertheast, the .urban Midwest, and an Indian 
Reservations in the West (See Table 1). The SA program annually 
serves approximately 6,203 children in over 250 classrooms. • The 
chi tore*, "Black id0l) 9 White (105), Indian {S%) 9 axi Puerto Rican 
(5*), share or^e fundamental commonality whicb has been clearly 
stated- by D^Bushell, Sr. (1974, p. 130), the originator of the 
Bk^rggrai*: , •'all of thep share a level of poverty that predicts 
'that tbeSf wijll fail in school unless we are ail e to -make sofce very 
basitf cfi^es." Bushel! f s claim that many of these children will 
"faij in school ■ refers to the fact that in irftny of the spools in 
which the^UpTSg^^ 13 operates, *more than 20% of the children who 
are in the fifth grade^ir-nqt^ve beginning reading skills* (1974, 
p. 130). 

_ Behavior Anal ysis ha s been called a new strategy for education 
which, combines traditional and innovative educational .techniques 
to provide a unique type of learning Opportunity for young chil- 
dren. While iiany of the basic principles of the model were drawn 
frtra modem learning theory, the BA progran^^s bas^d on the prin- 
ciples and procedures of the brauch of psychology known as applied 
behavior analysis which focuses on developing and implementing 



empirically-based solutions ta socially-significant problens. 

5eh3Yion Analysis conbioes aspects of team teaching, snail 
group instrjjb^on, non-graded classrooms, individualized instruc- 
tion, prograiaaed instruction, feconrended curriculum*, a token 
re i nf crcement^jrs tern , an individualized curriculum targeting* pro- 
cedure with rSiular progress reports on each child's -academic pro- 
gress, and p^ntal participation {Bushell & £am^^974). The pro- 
gram emphasizes the teaching of basic academic. skills s&sh as read- 
ing, arithmetic, handwriting, and spelling* Each classroom 
staffed by a teaching team consisting of a certified teacher who 
leads th£ team arc^is primarily responsible for riding instruction, 
a paraprofessiooa^eacher's aide who guides the ari t^etic^Jnst^ 
tjon, and oae to parent aides who are responsible for spelling, 
tendwritimj, and 3^^wwal tutoring. 

'The curriculi^&^ials recxranended for use in BA classroom 
were selected becauWsdhey create a favorable learning situation 
for each child ahd^ontribute to reaching specified and measurable 
instructional objectives. In other words, curriculum naterials 
were selected which best fit the classroom Instructional approach 
and the program goals. 
Program Evaluation 

The Behavior Analysis model is supported by a nwlti face ted 
evaluation strategy, predicated upon four systematic process and* 
outcoae evaluation techniques- As show in Figure 1, the four basic 
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evaluation strategies are:, classroom observation, e&*£inuous pro- 
gress assessment, annual achievement testing, and an annual con- 
surer evaluation. • Each serves a unique function, is tzsed on a 
continuous and regular basis, and is designed to eofspfeaent the 
others. Table 2 presents a suroary description of the key pararo- 
eters of the four evaluation strategies. 
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The effectiveness of any social service program Is nost ac- 

» * 

curately judged in -terras of a combination of Pleasures vAich con- 
sider the various dimensions of "effectiveness, * e.g., test data, 
satisfaction d£ta, curriculum progress data and mod^l inpleraenta- 
tion data. The imilti faceted evaluation strategy of the BA program 
has been designed to consider all of these dimensions, ^his par- . 
ticular conibinatidn of process and outco:ne evaluations appears to 
be of tremendous educational significance in teras of the amount 
and type of data collected, the overall scope, and as a demonstra- 
tion of a code! of an in-house evaluation stragegy that is adapt- 
able to any educational settins 



The results of the BA progras over sevea^years, determined by 
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•the four evaluation strategies'* indicate tbat it has been i s 
cessful ^ijflucatiojsal intervention strategy- ' For example, t the 
achieveneai test data have qhns i stent! y demonstrated that the BST 
children significantly out/ perform. their non-3A counterparts; con- 
sumer/eval^ation data hare indicated a hitjh leve* of satisfaction. 

Kith the program purposes, procedures, and effectiveness. In3e- 

/ * 
pendent, outside -evaluations teve corroborated these data. Ctee- 

outside evaluation occluded that the BA program has been emi- 

- s J • ~ 

nerttly effective id improving academic functioning and is a valu- 
able educational Experience j(Levitt, 1974, p'.13). Evaluations 
have also indicated a high level of Jiodel implementation by the 
classroom teaching teams (Stallings and Kaskovotz, 1974). 

* The remaining sections of the paper will present a brief 
description of the four in-hause, evaluation strategies which 
together comprise the BA multifaceted evaluation system. 
Classroom Observation . 

A social intervention program as large and widely dispersed 
as the Behavior Analysis program faces & .considerable problem in 
maintaining the integrity of.the program n>odel at all of its sites. 

r 

This problem necessitated a procedure that would assure t^e replica 
tion of the jnodel at al| BA sites effectively, economically, and 
wio>>mnimiB3 supervision by .the University of Kansas. Thus*, a 





strategy caTKog for direct observation of tee classroom by trained 

- *V 

on-site personnel, sba^f trainers, was designed ^(fl^inpl evented. 



The purpose of this strategy is to assist in the ftraihing of ne* 

j w <- 

classroom personnel, assure node! -replicat|ona^ross all cla$s- 
rooms, and certify BA teachers. This strategy is based on ttfo sets 
of performance criteria. 

J/B^first set of performance criteria is the Instructional 
^&chin% Cri ter^^ttese nfne criteria specify the appropriate 
- instrvc'ifonal befiavior defined bf the BA code! . Table 3 lists the 
. nine criteria. Observations nade .by the staff trainers must result 
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{ i^a "yes* response to each of the criteria. Immediately following 
J~lbt observation, the trainer provides feedback to the teacher be- 
/glnningwith the positive aspects,ai*d the^, if necessary, the 
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areas ^iere corrective action is needed, 
- k second sfet^ef performance^criteria i§ used to evaluate the 
: /pperat^rt of the classroom motivation % (isken). system. Table 4 , 
-lYsts th& -eleven Exchange Teaching Criteria. If observations pro- 
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duce^yes* responses to each item, the trainer is assured that the 
motivational system is functioning properly.- 1 

If the teacher meets all of the criter-ia on 'both' performance 
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measures during the two observations, the performance is acknowl- 
edged by a Behavior /Analysis Certification form and ,a lette.r of 



coroiendation 
and to the dis, 
nel file* The 
servations are 



Copies gf theiletter are also sent to the principal 
rict offices /for placement in the teacher's person- 
centification procedure is repeated annually; ob- 



made as frequently as necessary to insure prpper 
model implementation. ^ * 

A set *>f training manuals written specifically for the BA 
trainers, describe the training, observation, and corrective pro- 
Gedures in detail^ (Nel son, Saudargas, and Jackson, 1974; Jackson, 

Minnis- Hazel, ana Saudargas, 1974). Generally, these manuals 

\ * • - / 

specify a three-step procedure used both to train new teachers, and 

to monitor the performance of experienced teachers- These steps - 

. are tor (a) provide a rationale for the desired performance, de- 

;j/ed bkt 

(b) model tb/ desired\behivior and allow the trainee to try it; 
and, (c) provide the trainee with feedback and praise- &> 
Thesye training and monitoring procedures are supported by 



jhavior, and set a criterion for the behavior; 



fprmal 
\ * 



/nd infonrfal observations by a training consultant (District 



Advisor) fromyche University of Kansas- Qch BA site^is served by 
a. consultants who uses the telephone Niiax/5, and monthly visits to 



the site ftp monitor the overall model implementation and to insure 

7/ ' % . 
that thy local projed^ staff are receiving appropriate in-service 

sujipo// and -technical training- 



Despite a programs' rhetoric and intentions > actual class- 

* ■* * * 

room instruction may bear little resemblance to, the intended model. 
The systematic classroom observations used in the BA program pra- 
vide the needed quality control to insure model implementation* 
This type of .process* evaluation makes ii possible for local train- 
ing personnel to immediately begin to remediate* problems as they • 
we observed'. 

Continuous Progress Assessment 

The most commonly used evaluative instrument in classrooms is" 

the standardized achievement t^st. These tests", prepared by spe-* 

* ^ • 

cialists,- arp administered'to "provide information abput how a 

♦ "/«■' \ 

child or. a group of children at any grade level compares with sirt- 

• . V ' " ' " * ' <* * ■ ' \ " 

ilar children throughout the nation in the skills which the insjtru- 

i ' - ' - . , 

ment measures" (Bushell, 1973, p. 37). The results of standardized 

*>' V/ # ; ' * 

achievement tests, are useful in making administrative decision^, in 

1 * » 

generally identifying gfross weaknesses and outstanding performance, 

"a* 

and in making .general statements about group performance. However, 
"The difficulty is that they tell you nothing about why the results 
turn out as tfiey do, nor do they suggest hQy to change tfje results 
on future occasions" (Bushellv 1973, p.3$j. Although achieve- , 
ment tests^ do* serve a purpose, teachers need a diagnostic instru- 
ment that suggests treatments, - • - 

With this in mind, the BA ^program tunned, to a continuous pro- 
gress assessment' procedure whi£h utilizes' a computer-based- system 






for raonitorfffg' the academic progress of each, child and classroom 
toward daily, weekly, or biweekly goals. By -meeting the goals it 

is assured that the child will have completed a specified amount 

^ * • 

of curriculum materia] during the year. UriefJy, -this procedure 
is based ^n establishing a year-ena goal for/each studentT divid- 
ing* the^curripul urn irrto^srna^^ and establishing 

a critferion for the successful completion of each'segment. Peri- 

/ ... ^ a 

* odiCr curriculum- imbedded tests that measure^ comprehension assure 
^ tha?tft[je child masters each^nit before beg^ning the next one. 
■This procedure allows children to work individually at their, own ^ 
»ace toward their individual year-end gcJal. 

The continuous^pwg^ess assessment; strategy is known as the 
Behavior Analyst National Communication System (BANCS). The' sys- . 
tern;, a fully operational communications network consisting of 
data transmission and receiving devices located. in all BA school 



Insert Figure 2 abopt here 
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distriits which §re connected to.K.lL, allows the transmission of 
tfoo.k and page curriculum placement data between the projects and 
K.U. As shown in Figure 2, the curriculum* placement dat^ fYows 
from the" classroom through the communicatibns system to the com- ' 
putation center at K.U. The processed data are then sent back to 
the school district for the teacher to use*. The^entire process 



frtfcsiet^ collect ion in the classroom to receipt of the processed 

d3ta by the teacher is accomplished in approximately 3Gr tenrrs. 

■* - 

The codtinuous ppegrgss assessment strategy and ye2r-end tar- 
geting procedure which fay necessity is curriculum specific, begins 
by dividing each book into a series of equal steps. The equality 
of the steps is based on the time that it takes to complete a Step 
rather than on the number of pages in a 'step. l^f> result a 
uniform set of steps, which vary in number of pages, each. of which 
takes no longer to complete than the other. Figure 3 show^the 
expected entry and exit level by grade fevel and the expected rate 
of step completion per' week for the reading currietf^um. 



Insert Figurr^ about here 



Setting year-end goals for one grade produces an entering 
placement position for th#' next.. The dif/erence between the enter- 
ing and "exit steps is the'number of steps 'to be completed during 
each year. For kindergarten, using reading curriculum "A," as 
shown in Figure 3, this procedure results in a rate of .528 steps 
per week; For first grade, .473; second grade, .417; and third 
grade, .407 steps per week. - m 

If all children entered at the saoe level and 'made the same 
• f « 

amount of>prfogress each week, then planning to this point would 
be sufficient. However, it is inappropriate to expect all chil- 
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ds&n to inake the 1 ssm? progress a^a to -be at the sane step by the 

end of the jear. It is appropriate though, to expect that each 

child be given the opportunity to uake as nrjch prosress as possi£l 

and be assisted in doing so. Kh3t 1s retired, then , are indi- 

vidua! izad progress targets that are adjusted each week depending 

* • 
tfpon how well the child did the previous week. At this point the 

computer becomes an invaloable tool because it can be programmed 

to analyze' the current curriculum position of gach child, relate 

it 'to previous progress, check the year-end §pal, and develop a 

new weekly target! Since a child's entering placement could be 

above, slightly below, or drastically below the expected entry 

level ,• different targeting strategies are required. 

There are three target options. Option A Is the priiaacy 

targeting strategy and singly subtracts the entry level, frora the 

previously computed year-end goal and divides the -difference by 

the number of weeks regaining in the school year* Children gen- 

« - 
erally begin working u*der. this option unless, their records indi- 

cate a low rate of progress." 

Option 0 is $ target plan which spreads the expected progress 

requirement over two years. It is directed at those students 

whose current placement is found to be below the expected place- 

4 « 

raent by more than half the mcsber of weeks regaining in the year. 
Under Option ft, the expected rate of progress is extended over a, 
two*-yea'r period: ?he new rate is less than the child was expected 



to do under Option A but wore than his current rate of progress* 
The philosophy here is .that being behind is no excuse for staying^ 
behiod. If the -child progresses satisfactorily at the new rate, 
he will reach the>expected yezr-ertd 002 1 for the subsecp^nt grade 
the following year, if the child's performance (Joes not* tup rove, 
the teacher can choose to place the child under a special option 
{explained below) whereby she selects the number of pages per day 
that sne wants the child to read. Options A and 0 are called the 
automatic options because the computer selects them. It is pos- 
sible rpr a student to begin on Option A, slide to B, and nove' 
back to A by the end of the year. 

The third option, Option 0, is a teacher selected option ^nd 
is available for teachers to use generally with children having 
extreffie difficulty; however, it way also be used for a child whose 
reading limits seeni boundless- Under this option, any number of 
pages can be selected as a target which then remains constant 
throughout each targeting period until the teacher initiates a 
change. 

The ioforaation or feedback report received by each teacher 
is output from the computer and immediately teletyped back to the 
district for classroom distribution- The report is essentially a 
continuously updated classroom roster showing the nud^r of ab- 
sences for each child, current dxrriculua placement, the book and 
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page progress since the last repsn, the number of pages. that had 

been targeted for tfte child during the previous target period and 

the book end page placement {target} for the next target period. 

The feedback report is a diagnostic instrument that suggests 

» 

both group and individual treatments^ For example, it is indis- 
pensable in a cVassrooa in which the instruction is individualized. 
The report, showing the step that each ohild is working on, provides 
a recurring metric of the spread of the class across the steps} if 
a teacher is not individualizing the Instruction, the children 
wilK^nd to bunch t?p on a few adjacent steps. -Al^o, if children 
spend too roach time on pertain steps, nay be*-a ,cue to redefine 
these units, or perhaps spend additional tine in a group instruc- 
tion going over some of the difficult concepts. 

These data point out children who are lagging bfehind the rest 
of the class; a slow rate of progress may raise doubts if the child 
has mastered the previous skills necessary to complete the step on 
which he is currently working. A child who is outpacing his class- 
.rates may be in the wrong instructional sequence. His rapid pro- 
gress is an indication that it may be possible for him to skip 
ahead to more advarrced units. 

* In BA classrooms, teachers are requested to plot and keep 

• * * «. 

track of the weekly .step progress of their children on a specially 
prepared step^progress chart. This information becomes a conven- 
ient measure of the overall class .progress. Also, a line is drawn 

16 
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on the chart frorc the expected entry ste? to the expected e*it 

step. By regjlarly checking the progress of the median child' 

against the slope connecting the entry and exit steps, the teacher 

can readily determine ho* the class is progressing toward their 

exit level for that grade. * 

BANCS provides a contmdsus record of class and indiv*du^ 

performance wnicta can be used to evaluate the effectiveness of 

various instructional procedures. It nas been noted that a most 

important feature of the feedback report is tnat rfcc 

"provides exactly th? kind of feedback to the teacher 
„ that helps refine and improve the structure and con- 
tent of an instructional sequence each time it is 
used. Rather than shifting from one technique to 
another on the basis of fad or the latest cute idea 
from the teacher's lounge, changes t&9 be based on 
data which progressively makes the entire sequence 
zore effective (Bushel!, 1973, p-42). 

National Achievement TestloTsttetegy (MATS) 

The second evaluation technique is the annual testing ^f BA 
w - * 

children and a comparable sample of non-BA children on a standard- 

ized achievement test- The lest results, used solely for^ipgranj 

evaluation, also allow a comparison to national test norms- The 

achievement test results -are not used to make individual decisions 

abouT^a child's abilities but only, to provide an indication of 

program effects by grade level, school district, and across the 

entire program. A unique aspect of this testing program is the 

definition. of each entering kindergarten group as a cohort which % 
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is tracked through successive years in the program. This proce- 
dure allocs van ojs comparisons of children in terms of the number 

> 

of years of continjojts expene^ce m toe program . 

It was, and still is the case that in twelve scribal districts 
served by the BA program / no fewer than seven different achieve- 
~ ment tests are used and, of course , are admnisteced at various 

times djrmg the year. Since the bf^errogrsn is widely dispersed, 
it Decaire obvious that u wajld oe impossibly to mace any definite 
achievement "statements gvven trie variety of instruments used. 
With the approval of each district we ^iastituted the National 
Achievement Testing Strategy 1 (^ATS) whicn uses t^e saire achievement 
t£st administered daring" the ninth irontn of eacn scnool year . | 
* * "Vhe Wide ftange- Achievement Test (dastick, Bijou* & Jastick^ 
^is the basis of NATS. The WRAT is a nationally known and widely 
used evaluative instrument with a set of national norms. One 



advantage of the WRAT is that its structure allows children fromi 
pre- kindergarten through fifth grade to take the identical sections 
of the test (Level IK This eliminates the necessity o£ artifi- 
cally-adljasting scores across grade level tests. From an economic * 
standpoint, the WRAT is relatively inexpensive and is easy to ad- 
minister. . . 

To insure reliable and consistent administration of the test 
across ?11 sitesCBA has developed its own tester training package 
consisting of instructional booklets and quizzes, videotapes, and 
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✓pract&e testing sessions (Konano^d, Skinner h tfeis, 1974 £ 1974a). 
A 3ocal testing supervisor arrf testers who generally have had some 
experience with the WRA7 are hired by each local project -to admin- 
ister and grade they tests: Jhree-ea > tester training sessions are _ 
conducted by the testing supervisor, and when necessary, by a *LU. 

. training tean. £ach tester is quizzed on the specially prepared 
administration and grading manuals, vigws the videotapes which 
denonstrate proper adrnn^st-rdlion aftd grading procedures, and ad- 
minsters the WRAT to a cfc)ld in a practice session^ Only those 

•testers who ireet the specified criteria are allowed to participate 
in the testing program. 

tffcon receipt of tfii completed %6 graded tests a t, K.U. the 

3A evaluation staff assigns an ideotrf ication number to .each sto- 
dent in order to iSeniify the data for future analyses. T^5t£ are 
also checked to see that all relevant demographic information *have 
been provided. Each test from each class is then thoroughly re- 
graded. If an error is confirmed the child's score is changed to ^ 
a *I4o Score" and is not included in subsequent analyses* Generally, 
the overall error rate is approximately SS. 

6rade equivalent scores, provided by the WRAT, ire calculated 
by determining the saean fraw score for each cohort which is then 
converted to a oean grade equivalent score. Project scores and a 
summary of national test scores -are distributed to each district^ 
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Consumer Satisfaction 

ffost programs^ are evaluated in terms of ho* well they ac- 
complish their intended purpose; if the evaluation is favorable, 
then the program is generally considered successful and beneficial. 
However, one might argue about the validity of a '•successful pro- 
gram" in the absence of any data which demonstrates the sattsfec- 

f 

tion of tn^ prograii) octfituners . In other words, it is plausible 
that a program has achieved its purpose but the consumers are dis- 
satisfied with the procedures used .or, some other aspect of the 
prograri; it is insufficient to believe that the ends justify th? 
means. The measurement of consumer satisfaction is as important- 
an. indicatoirof: "success" as any other evaluation measure. ^ 

The Behavior Ana lysis -.program tfces a survey o* consumer satis- 
faction to measure or evaluate the social validity of the program. 
In a general sense, social validity refers to the opinion or 
judgment of consunJers and re/evant observers about the purpose, 
procedures, and the effects of a social 'intervention program- Con- 
sumer satisfaction thep is a subjective measure of various aspects 
of a program gained by survey analysis. Such a survey cay be done 
as a' one- shot survey or, more preferably will be built-in to the 
progrss as a recurring evaluative procedure designed to provide 
continuous feedback to the program managers. 

"The purpose of the consumer satisfaction survey called the 
Annual Consumer Evaluation or ACE is to measure consumer satisfac- 
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tion with £he 3A instructional nodel, the services provided by the 
University of Kansas Support aod Development Center for Follow 
Tbroygh, as sponsor to the loc^f districts, and the services pro- 
vided by the BA staff at each local sits. Also, this device pro- 
vides consumers with a *soanding board" whereby they have the op- 

portunity to feedback their opinions and comments about the program, 

♦ 

Considers of the program ere: the children, their' parents, 
SA teachers and aides, BA trainers, Project Directors, school dis-~ 
trict administrators, and ncn-BA teachers and cfHldren in the 
schools in; which the program operates, la alT, there are seven 
categories of consumers or respondents. The ACE attempts to -sur- 
vey the total population of consumers— a total of about 10,000 
respondents! 

x * Questionnaires were designed for each respondent category. 
Each questionnaire, except for the childrens* consisted of standard 
background information questions and from thre^ th six program 
evaluation questions- 'Where possible all respondents were asked 
the same or similar questions. A space for comments was provided 
next to each question. *A Likert-type seven category response set 
from "completely satisfied" to "completely dissatisfied" .was used 
(Likert, 1932)- A response of "6" (satisfied) was cffcsen as the 
level of satisfaction that we would like to achieve- 

Questiormaires are mailed to the sites to be distributed to 
the respondents by a local project staff member, together with a, 

* * 
* # 
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set of instructions and a cover letter. The better notes that *e 
are interested in the respondents' open e^j honest opinions, and* 
tbat anonymity will be guaranteed. Tfce completes} questiortnaires 
are returned to K.U. for analysis. 

The children's questionnaires are handled differently. Since 
all BA children and a comparison group .of non-BA children are 
given an achievement test annually, the testers are instructed to 
ask the children five brief questions and record their responses 
Qn .their test. These data are collected asd analyzed after the 
tests are returned to K.U. 

We feel that the ACE is unique in that few school systems or 
-educational programs treasure consumer satisfaction and fewer yet 
• survey the total population of consumers along the three dimensions 
of purpose, procedures, and effects. Whether or not we can ulti- 
mately satisfy all the consumers remains to be seen. But a surwey 
such as the ACE provides the foundation to obtain the data neces- 
sary to correct any aspect of the .program that is'jadged unsatis- 
factory. It is important to emphasize this aspect of the consumer 
satisfaction survey, that is, that the data derived from the sur- 
veys are used .to improve unsatisfactory aspects of the program. 
Conclusion v v- 

Since its beginning the overriding goal of the Behavior Anal- 
ysis program has been to provide tfie children sewed .with an edu- 

m 

cational program that would insure a level of functioning in the 



^ ; * feasic academic skills conparable to non-posr, "middle 'class* chil- 

A > ^ , / " £ren. Succinctly, this, goal has been to provide a year's worth of 



/ .- " 

# 



academic achievement for each year in school . 

The success of the BA -program is in large part due to the 
systematic and comprehensive evaluation strategy which allows a 
regular and recurring flow of data used to monitor various aspects 
of the program^ and as a basis to initiate change, in an era of 
uncertainty about the most effective a^d efficient way to educate 
children, and with a heightened need to seH-. out the effective 
from the ineffective models, the need for comprehensive, data- 
based evaluation is paramount*, 

— ProyranrevaTuation is a complex and unsettled "art." Because 
of this, there are those who argue against any rigorous form of 
>~ v - evaluation— they would, so to speak, throw out the baby with the 
bath watqr. The UA program is well -aware of the man^pitfalls in- 
* yolved in evaluation but strongly believes there is no other way 
tojjauge program effectiveness; in the absence of, or with in- 
complete evaluations, no questions will ever be answered. 
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Figure Captions 

figure 1. The four evaluation strategies of tke Behavior 
Anal /sis program.- *&&r 

Figure 2. The p3th traveled by th£*3&NCS dat~ froaf classroora 
to K.U. and back to classroqnu Dotted lines indicate path of raw 
^iata; the solid lines, processed data; ao3 the solid, vavy lines 
indicate a personal contact rather than an actual delivery of 
processed data. 

> Figure 3. Yearly target . slopes for reading curricula r A* 
froir entry level t? exit. : level .(year-end goal}. 
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TABLE 1 



BEHAVIOR ANALYSIS . V . 

FOLLOW TKROUSH PROJECTS 18 OPERATION 

1975-76 



! LOCATION i 


SCHOOLS 


i 


1 

"CLASSES 
fK-3) i 


1 

CHILDREN ! 




1 , 


1 

i 

*8R0NX - • | 


>2' 


* 




19 


580 


"*HS?I RESERVATION, ARIZONA 


5 






21 


298 


INDIANAPOLIS, INDIANA 


4 






T5 . 


385 


KANSAS CITY, MISSOURI 


4 






25 


533 . 


LOUISVILLE, KENTUCKY 


4- 






21 


624 . 


KERIDIAH.i ILLINOIS 


2 






19 


3S6 


NORTHERN WyO<NE, MONTANA 


3 






11 


271 


PITTSFIELD, MASSACHUSETTS 


2 






8 


157 


♦PHILADELPHIA, PENNSYLVANIA 
*PORTASEVILL£, MISSOURI V 


3 






47 


1324 


1 






16 


384 


♦TRENTON, NEW JERSEY 


8 






36 


' 849 


MAUKEGAft, ILLIN8IS 


1 






22 


. 495 


< 




t 




* 

4 




TOTALS 


39 

{ 




• 268 


6273 











*Prograa initiated in the fall of 1968. All other progress initiated- - 
the fall of 1969. 
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Tab 1 * 2 



SAFT PSOSR'X F* ALU ATI ON 



ClassrooE 
Observation 



Continuous 

Progress 
Assessitent 



Annual 
Achi evement 
Testing 



Annual 
Consunrer 
Evaluation 



System 



Tramisg 
and 
Monitoring 



BANCS 1 



j Type oi 
| Evaluation 

-» 


Data 
Collected 

i 1 


| . Type of ; 

Dsta j n 
1 Collected j Sl « 


Subj ect^ 


i 

i • > 
; Process 

* 

: 

Process 

i ! 


3i -Weekly 
Weekly 

- I 


1 
i 

Observation j - 
Data | 

i 

Curriculum 6,273 ; 

' Placements f 

j 


Teachers, 
Children 

Children 



KATS 2 i Outcome 



Process 

and 
Outcome 



tazrjtal 1 / , Achi erexrent ; 1 1 , 483 
j Test Scores 1 



Annually Question- \ 10,720 
i naires i 



Children * 



All Vrograa 
Consumers 



^Behavior Analysa s^jgationai Ccmncunication System 
^National Achievement Testing Strategy 
3 Annual Consumer Evaluation \ 
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Table £\ . 
Instructional Teeming Criteria* 

40 

1. 80S of the children are on- task. 

2. 100J of teacher's contacts are^-ta children who are on-task. 

3. 1O0S of teacher's contacts contain praise. 

4. 1005 of tokens given out are paired with praise. 

5. 90S of teacher's contacts that include proupts also contain 
descriptive praise and tokens. 

6. None of teacher's contacts are disapprovals. 

7. Tine-out procedures, if needed, are used appropriately. 
(See Bushel 1, 1973,* pp. 75-78). 

8. Four children in group, selected randomly; are working at 
802 accuracy. 

9. For a period of four weeks, the percentage of children *on- 
target" in the teacher's instructional groups nwst be SOS, 
as calculated by the computer. 

♦Abstracted fronj Kelson, Saudargas, and Jackson, 1974, p. 4. 



Table A 

Exchange Teaching Criteria* 

1. Back-up reinforcing activities are prepared and ready prior 
to the start of the instructional period. 

2. Prices and content of the back-ups vary at two exchange periods 
observed on the yfa* day. 

3. Prices vary for/each instructional group. 

4. Prices are tKe saiae for all children within each group. 

5. Children ^re free to choose any activity for Wff|fh they have 
enough tokens. 

6. Children with too few tokens s4t quietly during the exchange 
period. 

7. The first child in a group who js ready to exchange is a}- 
1 

lowed to do so without waiting for the other children to get 
ready. 

8. At least one back-up contributes to a relevant academic skill 

✓ * 

9. Adults participate in the back-up activities. ^ 

10. Teachers give praise for appropriate play during exchange 
periods. 

11. At the end of the exchange, instruction begins with the first 
child who coraes to the -table. 

♦Abstracted froa Kelson* Saadargas, and JacVson, p. Si 



3'4 



